Discourse Characteristics of Test Takers’ Spoken and Written Responses

Among the most innovative features of the TOEFL® Internet-based test (TOEFL iBT) are the addition of a speaking measure and a revised writing measure consisting of two writing tasks. The speaking and writing measures include both independent and integrated tasks. Independent tasks require test takers to talk or write about their personal opinions or experiences. Integrated tasks require the use of more than one language skill. For example, one writing task requires test takers to read a short passage, listen to a brief lecture, and then write about how the information in the passage and lecture are related. The rationale for including tasks that require speaking and writing on TOEFL iBT is that these tasks directly assess test-takers’ abilities to use English to communicate in academic situations. An assumption underlying this rationale is that these abilities should be evident in the characteristics of the discourse the test takers produce when completing these tasks. Analyses of discourse characteristics provide an important source of validity evidence for the interpretation of TOEFL iBT scores. As new task types were being developed for TOEFL iBT, two groups of researchers investigated how the discourse characteristics of spoken and written responses varied with ability level and task type. Using responses to early prototypes of speaking tasks, Brown, Iwashita, McNamara, and O’Hagan (2005) analyzed the nature of the discourse characterizing speech samples at different levels of ability and for different task types. These researchers first identified the categories of speech qualities that English as Second Language (ESL) experts attended to when rating the responses. Brown et al. then analyzed aspects of test-taker discourse in the responses associated with the experts’ categories, such as linguistic resources, phonology, fluency, and content. Many of the features— grammatical accuracy and complexity, vocabulary, pronunciation, speech rate, amount of content—in each of the categories were found to vary with score level to some degree, and a lesser amount with task type. When compared with performances on independent tasks, performances on integrated tasks had a more complex schematic structure and included more sophisticated vocabulary, but were less fluent. Their finding of more complex schematic structure for responses to integrated tasks was consistent with the rationale for including such tasks. In a similar manner, Cumming, Kantor, Baba, Eouanzoui, Erdosy, and James (2005) reported important differences in the discourse characteristics of written responses related to proficiency level as well as task types. Greater writing proficiency was associated with longer responses, greater lexical sophistication, syntactic complexity, and grammatical accuracy. Cumming et al. also compared the qualities of responses to the two task types. Responses to independent tasks were longer, had a more fully developed argument structure, relied on the self as a source of evidence, and used declarations—statements of personal opinions or facts. Responses to the integrated tasks had greater lexical sophistication and syntactic complexity, relied on the source materials for information, and used paraphrasing and summarization. To sum up, these researchers documented how the discourse characteristics of spoken and written responses to prototype test tasks varied with proficiency level and with task type. These studies have two important implications. First, holistic scores on speaking and writing tasks can be verified and anchored empirically through the analysis of important discourse qualities that differentiate proficiency levels and task types. Second, the introduction of integrated tasks on TOEFL iBT would have the desired effect of broadening the constructs assessed by the speaking and writing measures. The integrated tasks provided test takers an opportunity to demonstrate discourse qualities that were different from those demonstrated on independent tasks.